Skip to content

fix(concurrency): centralize write transactions with bounded SQLITE_BUSY retry - #3

Merged
codeo1io merged 2 commits into
external-memory-backendfrom
external-memory-backend-concurrency
Jun 20, 2026
Merged

fix(concurrency): centralize write transactions with bounded SQLITE_BUSY retry#3
codeo1io merged 2 commits into
external-memory-backendfrom
external-memory-backend-concurrency

Conversation

@codeo1io

Copy link
Copy Markdown
Owner

Problem

The plugin issues 100+ write transactions across 40+ files against ONE shared SQLite file (context.db), opened by multiple processes (OpenCode + Pi, or two OpenCode instances). Each hand-rolled BEGIN IMMEDIATE / COMMIT / finally-ROLLBACK block contends for the single WAL writer lock. Three problems:

  1. Inconsistent SQLITE_BUSY handling: when a sibling process held the writer lock past busy_timeout, the thrown SQLITE_BUSY propagated up and surfaced as failed to load plugin ... database is locked or Hit a transient issue comparting history this turn. Some sites retried, some swallowed, some propagated — unpredictable under real multi-process load.

  2. Duplicated transaction plumbing: every BEGIN IMMEDIATE site reimplemented the same try/commit/finally-rollback pattern, slightly differently, sometimes without the rollback path. Bugs leaked in at the edges.

  3. No retry on transient contention: a long-running sibling transaction (large migration, dreamer run, Channel-2 bulk delivery) made the plugin disable itself for the run instead of waiting.

Fix

A single shared runWriteTransaction(db, body) helper (packages/plugin/src/shared/write-transaction.ts) that:

  • Wraps the BEGIN IMMEDIATE / COMMIT in a bounded SQLITE_BUSY retry loop (4 attempts, 100ms backoff) so transient cross-process contention makes us wait-and-retry instead of throwing.
  • Composition-safe: if called inside an existing db.transaction() or another runWriteTransaction, the body runs inline WITHOUT issuing a nested BEGIN (SQLite doesn't allow nested BEGIN; the outer transaction already holds the writer lock). Detected via the bun:sqlite inTransaction flag or the node:sqlite shim's isTransaction flag.
  • Centralizes one correct transaction plumbing pattern instead of 9 slightly-different copies.

Also adds runWriteTransactionAsync for async call sites (same retry + composition semantics, body may be async).

Sites converted (9 sync write paths)

File Functions
dreamer/lease.ts acquireLease, renewLease, releaseLease
git-commits/sweep-coordinator.ts acquireGitSweepLease, renewGitSweepLease, markGitSweepSuccessAndRelease
message-index.ts indexMessagesAfterOrdinal (cross-process FTS dedup)
workspaces.ts bumpEpochsForWorkspaceMembers, bumpEpochsForWorkspaceMemberSet
compartment-storage.ts replaceAllCompartmentStateAndBumpDepth, promoteRecompStaging
key-files/project-key-files.ts replaceAllKeyFiles
key-files/identify-key-files.ts commitKeyFilesUnderLease
hooks/compartment-runner-recomp.ts promoteRecompStagingWithM0Mutation
hooks/compartment-runner-incremental.ts historian publish path

inject-compartments.ts is intentionally NOT converted: its two BEGIN IMMEDIATE blocks have complex early-return-with-rollback-and-fallback-read patterns (Phase 3 materialization contention retry, soft-refresh cache replay) that don't map cleanly onto the shared helper without significant restructuring. The helper is still a net win across the 9 simpler sites.

Removed

Two duplicate local runImmediate functions (in dreamer/lease.ts and git-commits/sweep-coordinator.ts) and the local isInTransaction helper in workspaces.ts — all subsumed by the shared helper.

Verification

  • bun run typecheck (tsc --noEmit + scripts) passes
  • bun run lint (biome check) passes
  • 12 new unit tests for runWriteTransaction / runWriteTransactionAsync pass
  • All existing tests in converted files' suites pass (dreamer/lease, git-commits, compartment-storage-atomic, compartment-runner-recomp-fk, key-files, workspaces, storage-db, message-index)

Relationship to the cold-start fix

This complements the cold-start database is locked fix (db3213e5, also in this branch's history). That commit fixed the cold-open race (busy_timeout installed before the first read). This PR fixes the steady-state write contention (bounded retry on transient SQLITE_BUSY during normal operation). Together they eliminate both the startup and runtime sources of the database is locked plugin-disable failures.

Branch maintenance

This branch (external-memory-backend-concurrency) is rebased onto external-memory-backend each time upstream advances, so it always represents the delta of concurrency fixes on top of the latest external-memory-backend work.

Net: −199 lines / +40 lines across the 9 converted sites; +201 lines for the shared helper; +156 lines for its tests.

OpenCode Agent and others added 2 commits June 20, 2026 02:56
…USY retry

Replaces 9 hand-rolled BEGIN IMMEDIATE / COMMIT / finally-ROLLBACK blocks
with a single shared runWriteTransaction helper that adds a bounded
SQLITE_BUSY retry loop (4 attempts, 100ms backoff). This is a better
concurrency solution than the previous per-site pattern because:

1. Centralized retry policy: transient SQLITE_BUSY from cross-process WAL
   writer lock contention (sibling migration, dreamer run, Channel-2 bulk
   delivery) now waits-and-retries instead of propagating up and disabling
   Magic Context for the run. Previously each site either swallowed,
   propagated, or retried inconsistently.

2. Composition-safe: if called inside an existing db.transaction() or
   another runWriteTransaction, the body runs inline WITHOUT issuing a
   nested BEGIN (SQLite doesn't allow nested BEGIN; the outer transaction
   already holds the writer lock). Detected via the bun:sqlite
   inTransaction flag or the node:sqlite shim's isTransaction flag.

3. One correct transaction plumbing pattern instead of 9 slightly-different
   copies. Several sites were missing the ROLLBACK path or had subtle
   ordering bugs at the edges.

Sites converted (all sync write paths with hand-rolled BEGIN IMMEDIATE):
- dreamer/lease.ts (acquireLease, renewLease, releaseLease)
- git-commits/sweep-coordinator.ts (acquireGitSweepLease, renewGitSweepLease,
  markGitSweepSuccessAndRelease)
- message-index.ts (indexMessagesAfterOrdinal — cross-process FTS dedup)
- workspaces.ts (bumpEpochsForWorkspaceMembers, bumpEpochsForWorkspaceMemberSet)
- compartment-storage.ts (replaceAllCompartmentStateAndBumpDepth,
  promoteRecompStaging)
- key-files/project-key-files.ts (replaceAllKeyFiles)
- key-files/identify-key-files.ts (commitKeyFilesUnderLease)
- hooks/compartment-runner-recomp.ts (promoteRecompStagingWithM0Mutation)
- hooks/compartment-runner-incremental.ts (historian publish path)

inject-compartments.ts is intentionally NOT converted: its two BEGIN IMMEDIATE
blocks have complex early-return-with-rollback-and-fallback-read patterns
(Phase 3 materialization contention retry, soft-refresh cache replay) that
don't map cleanly onto the shared helper without significant restructuring.
The helper is still a net win across the 9 simpler sites.

Verification:
- bun run typecheck (tsc --noEmit + scripts) passes
- bun run lint (biome check) passes
- 12 new unit tests for runWriteTransaction/runWriteTransactionAsync pass
  (BEGIN/COMMIT, rollback on throw, transient BUSY retry, max-attempts
  give-up, nested-transaction composition, non-transient error passthrough)
- All existing tests in the converted files' test suites pass:
  dreamer/lease.test.ts (7), git-commits (19), compartment-storage-atomic
  (8), compartment-storage-v6, compartment-lease, compartment-runner-recomp-fk
  (4), compartment-runner-partial-recomp, key-files (14), workspaces,
  storage-db (11), message-index

Net: -199 lines / +40 lines across the 9 converted sites, +201 lines for
the shared helper + 156 lines for its tests.
…ifest

The external-memory-backend branch added the memory.external schema subtree
(provider/endpoint/banks/retain_sources/tags + recall sub-block) but did not
add a corresponding entry to the dashboard's config-field-coverage manifest.
The config-parity guard (config-parity.test.ts) enforces that every schema
leaf is either RENDERED by the ConfigEditor form or listed in
OMITTED_BY_DESIGN, so CI failed with 16 uncovered leaves.

memory.external is USER-config-only and operator-configured (no form widgets
exist for it), so the correct classification is OMITTED_BY_DESIGN with the
whole subtree covered by the 'memory.external' prefix.
@codeo1io
codeo1io merged commit 3688e25 into external-memory-backend Jun 20, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant